Ordinal treatment effect of mechanical thrombectomy

Author

Bradley Kolb

Comprehensive evaluation of randomized evidence for mechanical thrombectomy using bayesian implementation of random effects meta-analysis with ordinal data in stan using brms.

Dependencies

suppressMessages({
  library(tidyverse)
  library(brms)
  library(posterior)
  library(here)
  library(tinytable)
})

Data

all_dat <- readRDS("all_dat.rds")

Data description

Table 1 shows the data from the 27 randomized trials of thrombectomy. Trials are categorized by type of stroke treated.

Table 1: Ordinal outcomes from 24 thrombectomy RCTs
mRS at six months
type trial treatment zero one two three four five six
large angel thrombectomy 9 19 41 39 45 27 50
large angel medical 0 8 18 49 60 45 45
large rescue thrombectomy 2 3 9 17 33 18 18
large rescue medical 0 3 5 5 25 40 24
large select thrombectomy 2 9 25 31 27 15 68
large select medical 0 3 9 20 36 32 71
large tension thrombectomy 5 5 11 18 24 15 46
large tension medical 1 1 1 13 16 24 66
large tesla thrombectomy 6 8 8 23 30 23 53
large tesla medical 2 6 5 16 35 33 49
early escape thrombectomy 25 34 30 26 21 11 16
early escape medical 10 15 18 22 35 18 28
early extend thrombectomy 9 9 7 6 1 0 3
early extend medical 6 4 4 4 6 4 7
early mrclean thrombectomy 7 21 49 42 51 14 49
early mrclean medical 1 16 35 43 80 32 59
early piste thrombectomy 5 9 3 4 4 1 7
early piste medical 1 5 6 7 4 3 4
early resilient thrombectomy 9 13 17 24 14 7 27
early resilient medical 3 7 13 17 21 18 33
early revascat thrombectomy 7 18 20 19 8 12 19
early revascat medical 6 7 16 20 17 21 16
early swift thrombectomy 17 25 17 12 15 3 9
early swift medical 8 10 15 16 20 12 12
early therapy thrombectomy 3 10 6 8 15 2 6
early therapy medical 1 6 7 7 10 4 11
early thrace thrombectomy 31 39 36 25 34 11 24
early thrace medical 24 33 28 25 56 9 27
late dawn thrombectomy 10 24 18 14 14 8 20
late dawn medical 4 5 4 16 34 19 18
late defuse3 thrombectomy 9 15 17 14 17 7 13
late defuse3 medical 7 4 4 14 24 14 23
late mrcleanlate thrombectomy 23 31 46 28 31 33 61
late mrcleanlate medical 5 35 44 20 27 42 74
late positive thrombectomy 4 4 1 0 1 1 1
late positive medical 1 3 5 3 2 3 4
basilar attention thrombectomy 11 34 29 29 11 27 84
basilar attention medical 5 5 3 14 6 19 63
basilar baoche thrombectomy 7 20 17 8 10 15 34
basilar baoche medical 1 6 7 11 20 16 45
basilar basics thrombectomy 8 19 27 14 10 17 59
basilar basics medical 6 13 25 11 16 12 63
basilar best thrombectomy 4 11 7 6 5 11 22
basilar best medical 6 10 2 3 12 7 25
medium escapemevo thrombectomy 45 61 32 46 28 9 34
medium escapemevo medical 38 80 43 52 26 12 23
medium distal thrombectomy 36 58 59 47 26 3 42
medium distal medical 47 54 46 57 24 3 38
historical ims3 thrombectomy 53 69 55 71 64 20 83
historical ims3 medical 19 39 28 35 30 15 48
historical synthesis thrombectomy 22 33 21 37 32 10 26
historical synthesis medical 28 35 21 28 38 13 18
historical mrrescue thrombectomy 1 3 3 13 18 13 12
historical mrrescue medical 1 5 4 13 15 9 7

“Large” refers to anterior circulation strokes with a large infarct core. “Early” refers to anterior circulation strokes with a small core, treated in an early time window. “Late” refers to anterior circulation strokes with a small core, treated in an extended time window. “Basilar” refers to posterior circulation strokes due to vertebro-basilar occlusion. “Medium” refers to anterior circulation strokes due to medium vessel occlusion.

Data exploration

Figure 1 plots the data from Table 1.

Figure 1: Distribution of ordinal values by treatment group for all thrombectomy RCTs

Model

Model description

We use brms to define and fit an ordinal meta-regression in the probabilistic programming language stan. Specifically, we use an adjacent category ordinal regression with category specific treatment effects, correlated trial-specific varying intercepts and slopes, and a predictor term for stroke type.

Model summary

Summary of the model is shown as follows.

 Family: acat 
  Links: mu = logit; disc = identity 
Formula: ordinal_value ~ type + cs(treatment) + (1 + treatment | trial) 
   Data: all_dat_long (Number of observations: 7695) 
  Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup draws = 4000

Group-Level Effects: 
~trial (Number of levels: 27) 
                                     Estimate Est.Error l-95% CI u-95% CI Rhat
sd(Intercept)                            0.15      0.03     0.09     0.22 1.00
sd(treatmentthrombectomy)                0.13      0.03     0.09     0.19 1.00
cor(Intercept,treatmentthrombectomy)    -0.57      0.25    -0.90     0.06 1.01
                                     Bulk_ESS Tail_ESS
sd(Intercept)                            1195     1878
sd(treatmentthrombectomy)                1435     2346
cor(Intercept,treatmentthrombectomy)      738     1339

Population-Level Effects: 
                         Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
Intercept[1]                -1.03      0.10    -1.22    -0.83 1.00     2151
Intercept[2]                -0.39      0.09    -0.57    -0.21 1.00     1941
Intercept[3]                -0.60      0.09    -0.77    -0.43 1.00     2049
Intercept[4]                -0.53      0.08    -0.70    -0.37 1.00     1693
Intercept[5]                 0.14      0.08    -0.03     0.31 1.00     1959
Intercept[6]                -0.81      0.08    -0.97    -0.65 1.00     1609
typeearly                   -0.36      0.07    -0.50    -0.23 1.00     1368
typelate                    -0.30      0.08    -0.46    -0.15 1.00     1844
typebasilar                 -0.09      0.08    -0.25     0.06 1.00     1625
typemedium                  -0.53      0.14    -0.79    -0.26 1.00      943
typehistorical              -0.31      0.11    -0.54    -0.09 1.01      936
treatmentthrombectomy[1]    -0.21      0.11    -0.42    -0.01 1.00     3488
treatmentthrombectomy[2]    -0.09      0.09    -0.27     0.10 1.00     3039
treatmentthrombectomy[3]    -0.31      0.09    -0.48    -0.13 1.00     3461
treatmentthrombectomy[4]    -0.34      0.09    -0.52    -0.17 1.00     3141
treatmentthrombectomy[5]    -0.21      0.10    -0.40    -0.02 1.00     3472
treatmentthrombectomy[6]     0.36      0.09     0.18     0.54 1.00     3282
                         Tail_ESS
Intercept[1]                 2747
Intercept[2]                 2918
Intercept[3]                 2557
Intercept[4]                 2279
Intercept[5]                 2236
Intercept[6]                 2191
typeearly                    1952
typelate                     2256
typebasilar                  2545
typemedium                   1951
typehistorical               1389
treatmentthrombectomy[1]     3015
treatmentthrombectomy[2]     2420
treatmentthrombectomy[3]     2633
treatmentthrombectomy[4]     2792
treatmentthrombectomy[5]     3087
treatmentthrombectomy[6]     2985

Family Specific Parameters: 
     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
disc     1.00      0.00     1.00     1.00   NA       NA       NA

Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

Model diagnostics

Because the model is fit to the data using MCMC, it is important assess for any errors during the sampling process.

Figure 2 shows the trace plots for each parameter of the model. Trace plots show good exploration of the parameter space.

Figure 2: Trace plots for each parameter

There were no divergences during model fit. Figure 3 shows rhat values all less than 1.05.

Figure 3: Rhat values for each parameter

Figure 4 shows that the estimated ratio of effective sample sizes to total sample sizes was greater than 0.1 for all parameters.

Figure 4: Estimated ratio of effective to total sample size for each parameter

Model checks

In addition to verifying that there were no technical problems with the MCMC algorithm when fitting the model, we also need to check for evidence of over-fitting or under-fitting to the data.

Figure 5 is a graphical posterior predictive check of the model fit to the observed data. This can be thought of as the bayesian analogue of a chi-squared goodness-of-fit test. The posterior predictive distributions of the model (yrep) all include the observed values (y). There is no obvious evidence of misfit of model predictions to data.

Figure 5: Graphical posterior predictive check

We can also check the predictive performance of the model using leave-one-out cross-validation.


Computed from 4000 by 7695 log-likelihood matrix

         Estimate   SE
elpd_loo -14045.5 39.8
p_loo        54.9  0.7
looic     28090.9 79.6
------
Monte Carlo SE of elpd_loo is 0.1.

All Pareto k estimates are good (k < 0.5).
See help('pareto-k-diagnostic') for details.

There are no signs of misspecification: estimated parameters closely match actual parameters, Monte Carlo standard error of the expected log posterior density is low, and all Pareto K estimates are less than 0.5.

Results

Ordinal treatment effect of thrombectomy across stroke types

The model considers a hypothetical patient enrolled in a new randomized trial of thrombectomy and uses the totality of observed data from previous trials to generate expected probabilities for each mRS at six months if the patient were enrolled in the medical arm of the trial, as well as the change in probability (either increase or decrease) if the patient was instead enrolled in the treatment arm of the trial. We refer to these changes as the ordinal treatment effect.

Large stroke

Table 2 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new large core stroke trial.

Table 2: Expected ordinal probabilities in medical arm of hypothetical new large stroke trial
estimate
mRS mean low high
0 1.6 0.9 2.7
1 4.4 2.8 6.5
2 6.5 4.6 8.6
3 11.7 9.5 14.0
4 19.8 18.0 21.6
5 17.2 15.5 19.0
6 38.8 31.5 45.6

Table 3 shows the expected ordinal treatment effect of thrombectomy in this trial.

Table 3: Expected ordinal treatment effect of thrombectomy in new large stroke trial. Positive numbers reflect an absolute increase in probability of the mRS category. Negative numbers reflect an absolute decrease.
estimate
mRS mean low high
0 1.8 0.9 3.0
1 3.2 1.7 4.9
2 3.8 2.2 5.4
3 1.9 0.2 3.6
4 -3.4 -5.5 -1.4
5 -5.7 -7.6 -3.8
6 -1.6 -5.8 2.8

Small stroke, early window

Table 4 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke early window trial.

Table 4: Expected ordinal probabilities in medical arm of hypothetical new small stroke early window trial
estimate
mRS mean low high
0 6.6 4.7 9.0
1 12.8 10.1 15.8
2 13.2 11.2 15.1
3 16.7 15.2 18.1
4 19.8 18.2 21.4
5 12.0 10.1 14.0
6 18.9 14.7 23.4

Table 3 shows the expected ordinal treatment effect of thrombectomy in this trial.

Table 5: Expected ordinal treatment effect of thrombectomy in new small stroke early window trial. Positive numbers reflect an absolute increase in probability of the mRS category. Negative numbers reflect an absolute decrease.
estimate
mRS mean low high
0 5.2 3.2 7.5
1 5.7 3.3 8.2
2 4.4 2.4 6.3
3 -0.4 -2.3 1.5
4 -6.0 -8.0 -4.1
5 -5.3 -6.9 -3.7
6 -3.7 -6.7 -0.8

Small stroke, late window

Table 6 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke late window trial.

Table 6: Expected ordinal probabilities in medical arm of hypothetical new small stroke late window trial
estimate
mRS mean low high
0 5.4 3.2 8.4
1 11.0 7.5 15.1
2 12.0 9.3 14.7
3 16.1 14.2 17.8
4 20.3 18.5 21.9
5 13.1 10.4 15.7
6 22.0 15.4 29.0

Table 7 shows the expected ordinal treatment effect of thrombectomy in this trial.

Table 7: Expected ordinal treatment effect of thrombectomy in new small stroke late window trial. Positive numbers reflect an absolute increase in probability of the mRS category. Negative numbers reflect an absolute decrease.
estimate
mRS mean low high
0 4.6 2.5 7.0
1 5.5 3.2 7.9
2 4.5 2.6 6.4
3 0.2 -1.9 2.3
4 -5.7 -7.7 -3.7
5 -5.5 -7.2 -3.8
6 -3.6 -6.9 -0.4

Basilar stroke

Table 8 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke late window trial.

Table 8: Expected ordinal probabilities in medical arm of hypothetical new basilar stroke trial
estimate
mRS mean low high
0 2.4 1.3 4.0
1 6.1 3.7 8.9
2 8.1 5.7 10.6
3 13.3 10.9 15.5
4 20.5 18.9 22.1
5 16.3 14.1 18.2
6 33.3 26.2 41.0

Table 9 shows the expected ordinal treatment effect of thrombectomy in this trial.

Table 9: Expected ordinal treatment effect of thrombectomy in new basilar stroke trial. Positive numbers reflect an absolute increase in probability of the mRS category. Negative numbers reflect an absolute decrease.
estimate
mRS mean low high
0 2.5 1.3 4.2
1 4.0 2.2 6.0
2 4.2 2.6 6.0
3 1.6 -0.2 3.3
4 -4.2 -6.3 -2.1
5 -5.7 -7.6 -4.0
6 -2.5 -6.5 1.8

Medium stroke

Table 10 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new medium stroke trial.

Table 10: Expected ordinal probabilities in medical arm of hypothetical new medium stroke trial
estimate
mRS mean low high
0 11.4 5.4 18.9
1 18.2 11.1 25.2
2 15.7 11.9 18.3
3 16.8 14.9 18.3
4 17.0 12.5 20.7
5 8.9 5.2 13.2
6 12.1 5.6 21.6

Table 11 shows the expected ordinal treatment effect of thrombectomy in this trial.

Table 11: Expected ordinal treatment effect of thrombectomy in new medium stroke trial. Positive numbers reflect an absolute increase in probability of the mRS category. Negative numbers reflect an absolute decrease.
estimate
mRS mean low high
0 6.8 3.6 10.5
1 5.5 2.5 8.4
2 3.2 0.4 5.7
3 -1.9 -4.2 0.7
4 -6.2 -8.1 -4.4
5 -4.3 -6.2 -2.6
6 -3.1 -5.7 -1.0

Summary

Conditional on the observed data and the model used to analyze the data, thrombectomy is expected to reliably lower the probability of bad outcomes and raise the probability of good outcomes across all stroke types. Figure 6 plots these estimates to show the trend graphically.

Figure 6: Expected ordinal treatment effect of thrombectomy in a hypothetical new trial

These treatment effects should be considered in the context of the expected probability of each outcome with medical treatment alone, as shown in Figure 7, which shows considerable variation by stroke type.

Figure 7: Expected ordinal probabilities in a new trial with medical treatment alone

Absolute dichotomous effect

Table 12 shows the implied dichotomous treatment effect estimate from this model.

Table 12: Effect of treatment on expected probability of mRS 0-2 in a hypothetical new trial
estimate
type mean low high
large 8.8 5.2 12.7
early 15.3 10.7 20.1
late 14.5 9.9 19.5
basilar 10.8 6.7 15.2
medium 15.5 10.8 20.5
historical 14.6 9.8 19.7

Figure 8 plots the estimates with the observed effect for comparison.

Figure 8: Effect of treatment on expected probability of mRS 0-2 in a hypothetical new trial. Mean observed effect plotted for comparison.

Figure 9 plots the cumulative effects.

Figure 9: Cumulative effect of treatment on expected probability of mRS 0-2 in a hypothetical new trial.

Sensitivity Analyses

Sensitivity to relative comparison versus absolute

Relative ordinal effect

Figure 10 shows the expected relative impact of thrombectomy on mRS in a new trial. Thrombectomy has a higher relative impact for large and basilar strokes because the base-rates of poor outcomes are high and the base-rates for good outcomes are low.

Figure 10: Expected relative impact of thrombectomy on mRS in new a trial

Relative dichotomous effect

Table 13 shows this effect in relative terms.

Table 13: Relative effect of treatment on expected probability of mRS 0-2 in a hypothetical new trial
estimate
type mean low high
large 1.7 1.4 2.1
early 1.5 1.3 1.7
late 1.5 1.3 1.8
basilar 1.7 1.4 2.0
medium 1.4 1.2 1.6
historical 1.5 1.3 1.8

Figure 11 plots the estimates with the observed effect for comparison.

Figure 11: Relative effect of treatment on expected probability of mRS 0-2 in a hypothetical new trial. Mean observed effect plotted for comparison.

Sensitivity to prior specification

With flat priors, there is no difference in out-of-sample performance.

         elpd_diff se_diff
omr_ip    0.0       0.0   
omr_flat -0.2       0.2

Figure 12 shows treatment effect estimate under flat prior assumption, which can be compared to Figure 6, the estimates under the main model specification.

Figure 12: Expected impact of thrombectomy on mRS in new a trial, flat priors

The results are virtually identical. Conclusion: reported results are relatively insensitive to prior choice.

Sensitivity to adjacent category parameterization

The main model assumes the adjacent category paramerization of ordinal regression, which allows for the modelling of category specific effects. An alternative parameterization of ordinal regression, referred to as cumulative ordinal regression, makes the proportional odds assumption.

        elpd_diff se_diff
omr_ip    0.0       0.0  
omr_cum -37.8       9.1

In addition to making the questionable assumption that the treatment effect of thrombectomy is constant across all mRS categories, this model also results in worse out-of-sample performance.

Sensitivity to meta-regression

Dropping predictors for stroke type does not result in decreased out-of-sample predictive performance.

       elpd_diff se_diff
omr_ip  0.0       0.0   
oma    -0.8       1.7

This implies that the ordinal treatment effect of thrombectomy is relatively consistent across stroke types, at least for this data set.